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ABSTRACT: Massive open online courses (MOOCs) are being used across the world to provide 
millions of learners with access to education. Many who begin these courses complete them 
successfully, or to their own satisfaction, but the high numbers who do not finish remain a subject 
of concern. In 2013, a team from Stanford University analyzed engagement patterns on three 
MOOCs run on the Coursera platform. They found four distinct patterns of engagement that 
emerged from MOOCs based on videos and assessments. Subsequent studies on the FutureLearn 
platform, which is underpinned by social-constructivist pedagogy, indicate that patterns of 
engagement in these massive learning environments may be influenced by decisions about 
pedagogy and learning design. This paper reports on two of these studies of learner engagement 
with FutureLearn courses. Study One first tries, not wholly successfully, to replicate the findings of 
the Coursera study in a new context. It then uses the same methodological approach to identify 
patterns of learner engagement on the FutureLearn platform, and explores how these patterns 
may have been influenced by pedagogy and elements of learning design. Study Two investigates 
whether these patterns of engagement are stable on subsequent presentations of the same 
courses. Two patterns are found consistently in this and other work: samplers who visit briefly, 
and completers who fully engage with the course. The paper concludes by exploring the 
implications for both research and practice. 


1 INTRODUCTION 

The generic name for MOOCs emphasizes their commonalities of scale (massive), economic/philosophical 
perspective (open), location (online), and structure (course). At the same time, it omits a key area of 
difference, their underlying pedagogy, which can also be described as their approach to teaching and 
learning and their understanding of how learning takes place. 

The original MOOCs, developed by Siemens and Downes from 2008 onwards, employed a connectivist 
approach to learning (Downes, 2012). This approach is signalled by references to them as cMOOCs. 
Connectivism considers learning to be social, technologically enhanced, distributed within a network and 
associated with the recognition and interpretation of patterns (Siemens, 2005). Knowledge is developed 
as a result of experience and the self-organizing nature of appropriately designed networks (Downes, 
2012 ). 

The cMOOCs were followed by xMOOCs. In this case, the defining letter "x" did not refer to a specific 
pedagogy, but to the role of these MOOCs as an extension to a previous offering. However, in many cases, 
this "extension" version took the essential elements of education to be content and assessment, with 
input from educators bundled as part of the content. This led to an instructivist approach to teaching and 
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learning in which "learning goals are predefined by an instructor, learning pathways structured by 
environment and learners have limited interactions with other learners" (Littlejohn, 2013). In Siemens' 
view, "cMOOCs focus on knowledge creation and generation whereas xMOOCs focus on knowledge 
duplication" (Siemens, 2012). 

This binary distinction between cMOOCs and xMOOCs was useful when the first xMOOCs appeared, but 
has become less relevant as new MOOC platforms and course developers have emerged. The FutureLearn 
platform, launched in 2013, employs a social-constructivist pedagogy, based on the Conversational 
Framework (Laurillard, 2002; Pask, 1976). This is a general theory of effective learning through 
conversations, with oneself and others, about the immediate world and about abstract concepts 
(Ferguson & Sharpies, 2014). To engage in successful conversations, all parties need access to a shared 
representation of the subject matter as well as tools for commenting, responding, and reflecting, and so 
these tools and shared representations formed part of the design of the FutureLearn platform. 

Although different types of MOOCs have different pedagogies, a problem that most of them experience 
is the large difference between numbers registering and numbers completing. Some of this drop-off can 
be explained in positive ways. MOOC registration can be regarded as similar to bookmarking a college 
website, sending off for a course prospectus, or placing a textbook temporarily in an online shopping 
basket — it does not necessarily represent a commitment to engage further with learning in that context. 
FutureLearn reports that around a third of those who register do not return to start the course. This 
suggests it is more useful to compare the number of learners starting a course with the number of learners 
who complete, rather than taking the number of people who register as the baseline. 

Up to a third of those who "drop out" may have clicked the "Register" button with little or no intention 
of completing — or even starting — the course. Another large group, characterized as "Samplers," appears 
to be made up of those who take a brief look at a course, decide it is not for them, and then leave (Kizilcec, 
Piech, & Schneider, 2013). Respondents to surveys associated with FutureLearn MOOCs have shared 
various reasons for participating, including trying out learning online, finding out more about MOOCs, or 
finding out more about a specific university. The learning objectives of these learners may be met without 
working through an entire course. 

Another perspective is offered by Downes, who points out that most media are not consumed in their 
entirety by any single individual: "It's actually very rare to find media of any sort that is intended to be 
consumed in its entirely. Most of the time, in most things, we pick and choose what is important 
to us. That is the normal mode of interacting with content [...] nobody thinks a library a failure if you don't 
read everything in the collection, or an author a failure if you don't read their entire corpus. And just so 
with MOOCs" (Downes, 2014). 

Despite these valid reasons and explanations for non-completion, the high dropout rates on most MOOCs 
still provide cause for concern. MOOCs are open, but "students seek not merely access, but access to 
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success" (Daniel, 2012) and it seems that many encounter not success but failure when they study these 
courses. The steep drop-off of the "Funnel of Participation" (Clow, 2013), which shows similar declines in 
engagement across different types of online learning sites, is not welcomed by all. 

Jordan's work (Jordan, 2014a) provides an overview of this issue, and her website (Jordan, 2014b) 
provides a dynamic view of the data about MOOC completion rates that has been made public. The site 
currently presents data relating to around 220 MOOCs based on a variety of different platforms. The site 
deals with the percentage of registered learners who complete a course (rather than the percentage of 
people who actually started the course who go on to complete). It shows that MOOCs typically report 
completion rates of around 15%. At the time of writing, no MOOC with more than 60,000 registered 
students had reported a completion rate of higher than 13%, and only three MOOCs had achieved a 
completion rate of over 40%. However, these high-performing outliers suggest that we should not be 
satisfied with MOOC current completion rates, because higher ones are possible. 

This variability in figures suggests that course context, course design and course pedagogy have an effect 
on retention, and thus on students' chances of success. 

1.1 Using Analytics to Investigate MOOCs 

Learning analytics are concerned with the use of trace data relating to learners and their contexts, for 
purposes of understanding and optimizing learning and the environments in which it occurs (SoLAR, 2011). 
They therefore offer a way of identifying factors that influence retention, enabling educators and platform 
providers to make changes to context, design, and pedagogy where appropriate. The large datasets 
generated by MOOC activity provide a strong basis for this type of approach. 

Kop and colleagues looked at engagement patterns on the PLENK2010 connectivist MOOC (Kop, Fournier, 
& Sui Fai, 2011). As this type of MOOC links a network of tools and resources, it is difficult to track all 
activity, but their conclusion was that 40-60 of the 1,641 learners registered on the course contributed 
actively on a regular basis, and that this engagement supported positive learning outcomes. The visible 
participation rate of others was much lower, indicating a consuming behaviour. This division between the 
Visible Contributors and the Consumers signalled that there are different ways of interacting on a MOOC. 
Milligan and colleagues investigated patterns of engagement in connectivist MOOCs and identified three 
distinct types of engagement: Active Participation, Passive Participation, and Lurking (Milligan, Littlejohn, 
& Margaryan, 2013). These classifications are strongly associated with connectivist pedagogy and to 
MOOCs that are structured to enable learners to construct knowledge together so are less relevant to 
courses in which key elements are pre-defined course content and assessment. 

In 2013, Kizilcec, Piech, & Schneider analyzed patterns of engagement and disengagement in three 
MOOCs on the Coursera platform. Two understandings underpinned this analysis. One was the view that 
"learning is a process of individual knowledge construction." The second was the view that the principal 
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features of these courses were video lectures and assessments. This analysis found four patterns of 
engagement with these courses: 

• Completing: These learners completed the majority of assessments. 

• Auditing: These learners watched most of the videos but completed assessments infrequently, if 
at all. 

• Disengaging: These learners completed assessments at the start of the course, then reduced their 
engagement. 

• Sampling These learners explored some course videos. 

As these clusters were found consistently across three MOOCs, each targeted at students working at a 
different educational level, it appeared plausible that they could be applied to other courses and that 
"MOOC designers can apply this simple and scalable categorization to target interventions and develop 
adaptive course features" (Kizilcec, Piech, & Schneider, 2013). This Coursera study has proved to be very 
influential. Not only is it highly cited within the academic literature but also the terms it uses are widely 
applied when describing and planning for MOOC engagement. 

However, the generalizability of these categories is qualified on the grounds that these classifications 
"would make sense in any MOOC that is based on videos and assessments" (Kizilcec, Piech, & Schneider, 
2013). This suggests that these categories may only be applicable to courses using pedagogies based on 
course content and on the use of either formative or summative assessment. 

In this paper, we investigate whether the same patterns of engagement are found in MOOCs that employ 
social constructivist pedagogy (Study One), or if other patterns of engagement apply. We also investigate 
whether patterns of learner engagement in MOOCs are stable across multiple presentations of the same 
course (Study Two). 

In social-constructivist MOOCs, knowledge is jointly constructed through conversation. Contributing to or 
reading discussion comments is therefore an important part of the learning process. In these cases, three 
elements should be taken into account: 1) active engagement with course content, 2) active engagement 
with course assessment, and 3) active engagement with course discussion. 

1.2 Replication 

Within the hard sciences, replication of previously published work is routine. Within the social sciences, it 
is unusual. Recent efforts to attempt to replicate key findings in social psychology have been controversial 
(Bohannon, 2014). There has been significant replication work in the Educational Data Mining field, 
particularly the body of work associated with the Pittsburgh Science of Learning Center's DataShop.^ 


^ http://www.learnlab.org/technologies/datashop/ 
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However, outside the context of cognitive tutors and similar fine-grained analysis, replication work in 
learning analytics is very unusual, as far as we are aware. As an empirical discipline, we believe that the 
field of learning analytics is well placed to use such rigorous approaches. To be clear, we regard the work 
of Kizilcec and colleagues as important and interesting, and in the studies reported here we are seeking 
to test how robust their findings are in a different context. 

2 FUTURELEARN DATASET 

In order to investigate patterns of engagement within MOOCs employing social-constructivist pedagogy, 
we used data provided by FutureLearn. This company, owned by The Open University, is currently in 
partnership with fifty universities, sixteen specialist organizations, and six centres of excellence to deliver 
free online courses. The company has developed a new MOOC platform based on scalable web technology 
and underpinned by a social-constructivist pedagogy (Sharpies & Ferguson, 2014). Each teaching element 
(step) is associated with a free-flowing discussion. These are intended to emulate a "water cooler 
discussion" about the immediate content. As with conversations around an office water cooler, people 
come and go, nobody is expected to engage throughout, but there can be a continuous sense of lively 
interaction. Typically, the discussion associated with any step on a FutureLearn MOOC will attract 
hundreds, thousands, or even tens of thousands of contributions, with "Like" and "Follow" options 
providing ways of navigating these. 

2.1 Study One: Identifying Patterns of Engagement 

Each FutureLearn partner institution has access to data related to courses they run. In Study One, designed 
to identify patterns of engagement in FutureLearn MOOCs, we therefore focused our attention on data 
from four MOOCs run by our institution. The Open University. The data on platform activity are not 
directly associated with demographic data (as discussed below), and so the figures for gender balance 
presented in Table 1 are taken from responses to the start-of-course survey for each course. These four 
MOOCs ran soon after the launch of FutureLearn, at a point when the majority of marketing was taking 
place within the UK, so the majority of participants on each course were based in the UK. 

The number of participants reported in Table 1 indicates the number of individuals — both educators and 
learners — who appeared in our Study One dataset because they were active on the platform after the 
start date of the course. People active before the start date were removed from the dataset entirely, 
because the only people with access at that point were educators or FutureLearn staff. In FutureLearn 
terms, a Fully Participating learner is one who marked a majority of the course steps complete and 
completed all the assessments. 
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Ta 

}le 1: Overview of dataset for Study One 


MOOCl 

MOOC2 

MOOC3 

MOOC4 

Subject area 

Physical Sciences 

Life Sciences 

Arts 

Business 

M 

51% 

39% 

32% 

35% 

F 

48% 

61% 

67% 

65% 

Participants 

5,069 

3,238 

16,118 

9,778 

Fully Participating 

1,548 

684 

3,616 

1,416 

Participation Rate 

31% 

21% 

22% 

14% 


Each of these MOOCs specified that no previous experience of studying the subject was required. 
However, the Arts MOOCS was recommended for learners aged 16+, due to the possibility that discussion 
content might include material that would be inappropriate for minors. Fully Participating learners on 
these MOOCs could gain a Statement of Participation for a fee, but no credits were awarded and 
participants were not required to complete assessments by a certain date. As a result, the "finished 
assessment on time" and "finished assessment late" classifications were of less relevance than on the 
Coursera study (Kizilcec, Piech, & Schneider, 2013). Coursera course content focuses on videos. While 
FutureLearn courses also include videos, they make use of substantial quantities of other learning 
material, predominantly discussion and text with pictures. 


The different context of Coursera and FutureLearn affected the raw data accessible to us as analysts. 
Kizilcec and colleagues were able to obtain full log data, assessment scores, demographic data, and survey 
responses, all at an individual learner level. This enabled them to report some interesting correlations. On 
early FutureLearn MOOCs, demographic, survey, and activity data were not linked by personal identifiers, 
so we had access to demographic and survey data only at a summary level for the courses we were 
researching. We were able to obtain individual-level activity data, which were anonymized before we 
received them. As well as assigning random IDs, this process partially aggregated the activity data, so we 
only had access to the date and time of a learner's first visit to a step, not the date or time of any 
subsequent visits. Graphical inspection of these data suggested that there were no gross differences in 
activity patterns compared to what one would expect in more traditional logfile analysis, although this 
change has had some effects, which we note where they arise. 

3 CLUSTERING 

3.1 Replicating the Method 

Our initial approach in Study One followed the one applied in the Coursera study as far as possible, in 
order to investigate whether previous findings could be replicated in a different context. That study 
adopted a methodology set out in Kizilcec, Piech, and Schneider (2013), designed to identify a small 
number of ways in which learners interact with MOOCs. 
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Kizilcec and his colleagues began by computing a description for individual learners of the way in which 
they engaged in each assessment period of the course (typically a week), and then applied clustering 
techniques to find subpopulations in these engagement descriptions. FutureLearn courses are divided into 
weeks and, although there is no compulsion to work through at the same pace as others, an overview of 
the data showed that activity did indeed follow a weekly cycle, spiking after the weekly course email was 
sent out each Monday (see Figure 1). Underlying the spikes, there was a general falling-off in activity over 
time, as would be expected in a MOOC (Clow, 2013). It is possible, however, that our measure of activity 
(time of first visit to each piece of content) masked later activity because re-visits to content visited earlier 
were not counted, and so Figure 1 is likely to over-state the fall-off in activity. 

We therefore followed the Coursera study and computed a description for each learner based on 
individual activity in a course week. Their "engagement descriptions" have six or eight elements, because 
each course ran for six or eight weeks. 


Daily Activity on MOOC1 



Mar - .‘.prO' ; 1' Ma, ' 

Date 

Figure 1: Activity on Study One, MOOCl. 

Dark red vertical lines show the date of the weekly course email. 

As in the Coursera study, students' activity in each week was assigned to one of four categories: 

• "T = on track" if they undertook the assessment on time 

• "B = behind" if they submitted the assessment late 

• "A = auditing" if they engaged with content but not with the assessment 

• "O = out" if they did not participate. 

Although FutureLearn students were not required to submit assessed work by a particular date, we 
retained these classifications and counted an assessment as submitted "late" if it was completed after the 
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end of the course week. For example, if a learner missed the first week of a course (out), submitted 
assessments on time in weeks 2-6 (on track), submitted week 7's assessment in week 8 (behind), and 
then engaged with content only in week 8 (auditing), their engagement profile during this section of the 
study would appear as [O, T, T, T, T, T, B, A]. 

Once we had created engagement profiles for each Study One learner in this way, we followed the method 
of Kizilcec's team and applied the /c-means clustering algorithm to partition the learners into a small 
number of groups. To cluster engagement patterns, a numerical value for the dissimilarity between them 
is required. There are several possibilities for this, but in this initial exploration, we followed their 
approach: we first assigned numerical values to each label (On Track = 3, Behind = 2, Auditing = 1, Out = 
0). We then calculated the Li norm for each engagement pattern and used that as the basis for one¬ 
dimensional /f-means clustering, thus minimizing the sum of the differences between individual patterns 
in each cluster. 


Learner activity on MOOC1 


On.Track 

Behind 

Auditing 

Out 


week 

Figure 2: Proportion of users on Study One MOOCl falling into each category, by week 

We also followed their practice in repeating clustering 100 times and selecting the solution with the 
highest likelihood, because k-means has random aspects. 

To attempt to replicate their findings, we focused on extracting four clusters. However, our clusters did 
not match those found in the Coursera study. The method produced two clusters that were very similar 
to the ones they found. We found a cluster strikingly similar to their "Completing" group: learners who 
completed almost all the assessments. We also found a cluster similar to their "Sampling" group: learners 
who visited only once or twice, and did not attempt any assessment. 
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The other two clusters did not match theirs so well. They found an "Auditing" cluster "who did 
assessments infrequently if at all and engaged instead by watching video lectures," and a "Disengaging" 
cluster "who did assessments at the beginning of the course but then have a marked decrease in 
engagement." Our two equivalent clusters did include some learners with patterns like that, but they also 
included many who did not fit those descriptions neatly, and there seemed to be significant overlap. This 
was reflected in the silhouette scores: they reported an average silhouette width of 0.8, but our data 
achieved only 0.67. (The closer to 1.0, the better clustered.) 

Selecting a number of clusters, k, to extract using /c-means is notoriously problematic, unless there is an 
unambiguous a priori rationale. A notable feature of /c-means clustering is that it will always generate k 
clusters regardless of whether another number would provide a better fit for the data. 

We therefore repeated the analysis using values for k that ranged from 3 to 8. We found that the 
silhouette width was at a minimum for k = 4, suggesting that this might be the least suitable number of 
clusters for our data. 

We were concerned that the one-dimensional approach was discarding potentially useful information 
about patterns of engagement before the clustering algorithm could use them, so we repeated the 
analysis again, this time running /c-means on the numeric engagement profiles directly, treating them as 
six- or eight-dimensional vectors. We explored the four-cluster solution in detail, and again found 
"Completing" and "Sampling" clusters, but no clear pattern in the other two. 

Again, we explored k from 3 to 8, which yielded silhouette widths rather lower than for the one¬ 
dimensional approach (around 0.4), and which decreased monotonically as k increased. 

3.2 Adapting the Method 

Having had mixed results from direct application of the method, we planned to adapt their method, 
making a minimal number of changes to reflect the context of our data. 

We developed a new classification, in order to reflect the importance of discussion in FutureLearn MOOCs. 
For each content week, students were assigned the value 1 if they viewed content, 2 if they posted a 
comment, 4 if they submitted the week's assessment late and 8 if they completed the final assessment 
before the end of the week in which it was set (i.e., early, or on time). These values were added up to give 
a total for each week. The possible scores that could be assigned to an individual learner in any one week 
are set out in Table 2 (below). 

The majority of comments on FutureLearn are associated with a content step, as study weeks typically 
include only one or two discussion steps. There is therefore no way to tell from our log data whether 
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learners had participated in the discussion by reading but not by adding a comment, so this option could 
not be coded. 


Table 2: Scoring method 

Score Interpretation 

1 only visited content (for example, video, audio, text) 

2 commented but visited no new content 

3 visited content and commented 

4 did the assessment late and did nothing else that week 

5 visited content and did the assessment late 

6 did the assessment late, commented, but visited no new content 

7 visited content, commented, late assessment 

8 assessment early or on time, but nothing else that week 

9 visited content and completed assessment early/on time 

10 assessment early or on time, commented, but visited no new content 


We used the /c-means algorithm to extract clusters from the engagement profiles directly, as a six- or 
eight-dimensional vector for each learner, to allow for the possibility of clustering by time of activity, as 
well as by total activity. The variation in dimensions here was because three of these MOOCs ran for eight 
weeks but MOOC2, the life sciences MOOC, ran for six weeks. 



Figure 3: Study One silhouette widths for k=2 to 8 (left) and 
scree plot of the within-groups sum of squares for k=l to 10 (right) 


Selecting a number of clusters to extract was very problematic. A wide variety of techniques was 
employed, almost all of which yielded unhelpful results, recommending either a minimal number of 
clusters (2) or a maximal number.^ 


^ Specifically, we applied all the techniques listed in this Stack Exchange post: 

http://stackoverflow.com/questions/15376075/cluster-analysis-in-r-determine-the-optimal-number-of- 

clusters/15376462 
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This in itself suggests that this entire approach to clustering was far from ideal. However, we wished to 
pursue the approach to explore whether the clusters nonetheless had the vivid face validity of the Kizilcec 
et al. paper. Manual inspection of the two-cluster solution suggested that this separated the learners into 
those who visited only briefly (Samplers — see below) and others. We wished to explore the other 
patterns of engagement in some detail. A maximal number of clusters would be uninformative and 
unwieldy analytically. 

Two methods supported a pragmatic decision to extract seven clusters. The mean silhouette width 
reached a local maximum at /c = 7 (see Figure 3, left). We also used a scree plot to visualize the total within- 
groups sum of squares (the measure of how closely the clusters group together that /c-means seeks to 
minimize). The method in using a scree plot is to identify a "kink" in the plot, which is somewhat 
subjective. In this case, the within-groups sum of squares dipped at k = 7 (see Figure 3, right). 

3.3 Analysis of Clusters 

Our cluster descriptions were developed by analysis of learner profiles on MOOCl. Like Kizilcec and 
colleagues, we wanted to ensure that the clusters made sense from an educational point of view, even 
though these descriptions were developed after the analysis had taken place. After developing the cluster 
profiles for MOOCl, we examined the other Study One MOOCs. With three exceptions, there was very 
close agreement between the clusters found. 

Two of the MOOCs were structurally different to the others. MOOCl, MOOC3, and MOOC4 ran for eight 
weeks, but MOOC2 ran for only six weeks. MOOCl, MOOC2, and MOOC4 included assessment every 
week, but MOOC3 included only three assessments. 

Despite these differences, five of the seven clusters were found in substantially identical form in all four 
MOOCs. MOOC2 and MOOC3 between them generated three clusters that did not match those developed 
on MOOCl, which are appended to the list as supplementary clusters. 

Cluster descriptions for Study One are given below, together with notes of where individual MOOCs 
differed. The clusters found across all four MOOCs are indicated with a *, as well as in the text. The 
proportion of learners in each cluster is shown in Table 3 (below). In all cases, the term "average" refers 
to the mean. We have provided typical engagement profiles for each cluster, which we derived by 
inspecting the profiles within each cluster manually, and selecting one that occurred very frequently. 

Cluster I: Samplers * 

Learners in this cluster visited, but only briefly. This cluster is essentially identical to Kizilcec et al.'s 
"Sampling" cluster. Samplers made up the largest cluster in all four MOOCs, accounting for 37%-39% of 
learners (56% on MOOC4). They typically visited about 5% of the course, with a few Samplers (ll%-24%) 
visiting only a single step, although only 1% of Samplers on MOOC3 did so. They were active in a very small 
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number of weeks, often including week 1, but not always — 25%-40% joined the course after the first 
week. Very few Samplers posted comments (6%-15%), and very few submitted any assessment, although 
the handful that did so typically did this in week 1. 

A typical engagement profile for this cluster shows individuals looking at content in Week 1 and not 
engaging in subsequent weeks: [1, 0, 0, 0, 0, 0, 0, 0], This cluster was highly stable across all MOOCs, and 
across most values of k. 

Cluster II: Strong Starters* 

These learners completed the first assessment of the course, but then dropped out. 

Strong Starters made up 8%-14% of learners. All of them submitted the first assignment, but then their 
engagement dropped off sharply, with very little activity after that. A little over a third of them posted 
comments (35%-38%), and those who did so did not post very many (1.7-4.0), except on MOOC3, where 
73% of learners posted an average of 13.7 comments. 

A typical engagement profile for this cluster shows them visiting content and submitting their assessment 
on time in Week 1, visiting content in Week 2, then disengaging: [9, 1, 0, 0, 0, 0, 0, 0], Again, this cluster 
was highly stable across all MOOCs and most values of k. 

Cluster III: Returners 

These learners completed the assessment in the first week, returned to do so again in the second week, 
and then dropped out. 

Returners made up 6%-8% of learners, with the exception of those studying MOOC3, where this cluster 
did not appear. This was almost certainly because there were only three assessments in that course, with 
at least one week between each, so this pattern of activity was not possible. Almost all the learners in this 
cluster (>97%) submitted the assessments for week 1 and week 2. This does not mean that they all visited 
in both weeks; some of them did the week 1 assessment late. No Returners explored all the course steps; 
the average amount of steps visited varied from 23% to 47%. After the first two weeks, there was very 
little activity indeed. 

A typical engagement profile for this cluster shows individuals viewing content and completing 
assessments on time in the first two weeks, but then disengaging: [9, 9, 0, 0, 0, 0, 0, 0]. This cluster showed 
some variability with varying k. 

Cluster IV: Mid-way Dropouts 

These learners completed three or four assessments, but dropped out about half-way through the course. 
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Mid-way Dropouts made up 6% of learners on MOOCl, and 7% of learners on MOOC4. This cluster did 
not appear for MOOC2 and MOOC3, because their unusual structure (shorter, fewer assessments) meant 
that there were not enough assessments other than the final one for them to have this profile. These 
learners visited about half of the course (47%, 59%), and roughly half of them posted comments (38%, 
49%), posting 6.3-6.5 comments on average. 

A typical engagement profile for this cluster shows individuals submitting assessments on time in the first 
three weeks, submitting late in the fourth week, viewing content in weeks five and six and then dropping 
out: [9, 9, 9, 4, 1, 1, 0, 0]. This cluster, like the previous one, showed some variability with varying k. 

Cluster V: Nearly There* 

These learners consistently completed assessments, but then dropped out just before the end of the 
course. 

Nearly There learners accounted for 5%-6% of learners on all four MOOCs. They typically visited over 
three-quarters of the course (72%-80%) and submitted assessments consistently (>90%) until week 5, and 
mostly on time (40%-75%), after which their activity declined steeply, and few completed the final 
assessment (3%-17%), none on time. Many of them posted comments (48%-65%), and those who did 
posted an average of 5.7-8.3 — except for MOOC3, where 80% of learners posted an average of 21.8 
comments each. 

A typical engagement profile for this cluster shows high engagement early on, viewing content and 
submitting on time in the first six weeks, and sometimes commenting as well, then submitting the 
assessment in week seven and ceasing to engage: [11, 11, 9, 11, 9, 9, 8, 0], This cluster appeared 
definitively for all four MOOCs, but was somewhat variable with varying k. 

Cluster VI: Late Completers* 

This cluster includes learners who completed the final assessment, and submitted most of the other 
assessments, but were either late or omitted some. 

Late Completers accounted for 6%-8% of learners, except on MOOC3, where only 0.2% of learners fell 
into this cluster. This was presumably because MOOC3 only included three assessment points, so there 
were fewer opportunities to get behind on the assessment. Each week, including the final week, more 
than 94% of this cluster submitted their assessment. The average proportion submitting late varied from 
16% to 59%. However, more than three-quarters submitted the final assessment on time (78%-90%). 
Fewer than half of these learners posted comments (40%-43%), apart from on MOOC3 where 76% did so, 
and those that posted made an average of 7.9-15.0 comments. 
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Atypical engagement profile for this cluster shows individuals visiting content and submitting assessments 
late in the first five weeks, and then viewing content and submitting assessments on time in the final 
week: [5, 5, 5, 5, 5, 9, 9, 9]. This cluster was fairly stable across all MOOCs, and across most values of k. 

Cluster VII: Keen Completers* 

This cluster consists of learners who completed the course diligently, engaging actively throughout. 

Keen Completers accounted for 7% to 13% of learners, apart from on MOOCl, where 23% of learners fell 
into this cluster. All learners in this cluster completed all the assessments, including the final one, and 
almost all of them on time (>80%). On average, the Keen Completers visited >90% of the course content. 
They were also assiduous commenters. About two-thirds of this cluster (68%-73%) contributed 20.8-24.4 
comments on average, although again MOOC3 stood out, with all learners commenting, posting an 
impressive average of 53.7 comments each. 

A typical engagement profile for this cluster shows individuals viewing comment and submitting 
assessment on time every week, and also commenting in several weeks: [11, 11, 9, 9, 11, 11, 9, 9]. This 
cluster was highly stable across all MOOCs, and across values of k. 

MOOC2 Supplementary Cluster 

Because it was shorter than the other MOOCs, the Mid-way Dropouts cluster was not found on this MOOC. 
Instead, a rather miscellaneous and hard-to-interpret cluster was generated, falling between Samplers 
and Strong Starters, and accounting for 13% of the learners. Sixty-seven percent of the cluster submitted 
the first assessment, but late, and many either visited regularly or left comments. 

MOOCS Supplementary Cluster 1 

MOOC3 had only three assessments, so the Returners and Mid-way Dropouts clusters were not found. 
The first alternative cluster was a substantial one, accounting for 20% of learners. It was very similar to 
the Samplers cluster in that the learners visited only briefly — however, all of them left comments. They 
left an average of 5.3 comments, which is high compared to the other MOOCs, but low for MOOC3. 

MOOCS Supplementary Cluster 2 

The second alternative cluster was much smaller, accounting for only 2% of MOOC3. These learners 
submitted the final assessment on time, but did not engage consistently across the course. In this cluster, 
44% submitted the first assessment, 22% the second. They visited on average 44% of the course. About 
half of these learners left comments (53%), at a rate of 10.1 each. 
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Table 3: Proportion of learners in each Study One cluster, by MOOC 


Cluster 

MOOCl 

MOOC2 

MOOCS 

MOOC4 

1 Samplers 

39% 

39% 

37% 

56% 

II Strong Starters 

11% 

14% 

8% 

10% 

III Returners 

6% 

8% 

- 

7% 

IV Mid-way Dropouts 

6% 

- 

- 

7% 

V Nearly There 

6% 

6% 

6% 

5% 

VI Late Completers 

8% 

7% 

0.2% 

6% 

VII Keen Completers 

23% 

13% 

7% 

9% 

MOOC2 Sup 

- 

13% 

- 

- 

MOOC3 Sup 1 

- 

- 

20% 

- 

MOOC3 Sup 2 

- 

- 

2% 

- 


4 DISCUSSION OF STUDY ONE 


Kizilcec and colleagues found four clusters within their data: learners who were completing, auditing, 
disengaging, and sampling (Kizilcec, Piech, & Schneider, 2013). They identified these clusters after 
structuring their data in a way that placed emphasis on learner interaction with content, assessment, and 
deadlines. As Section 3.1 showed, we found "Completing" and "Sampling" clusters, but the others were 
not found so clearly within the FutureLearn datasets. 


The socio-constructivist pedagogy of FutureLearn incorporates not only content and assessment within a 
course's learning design but also discussion. In addition, as credit is not currently available for FutureLearn 
courses, assessment is typically used formatively (to support learning) rather than summatively (to gauge 
how much has been learned). The results of our analysis differed from Kizilcec et al., which suggests that 
this different context may have influenced the results. Our approach included data about contribution to 
discussion as an input, and some of the clusters we found reflected particular patterns of participation in 
discussion. This suggests that a single approach is unlikely to be pedagogy-neutral. We suggest that 
opportunities for joint knowledge construction should be taken into account where they are available. 


The "Sampling" cluster identified in 2013 (Kizilcec, Piech, & Schneider, 2013) can be related to two clusters 
in the current study: the Samplers and the Strong Starters. The Samplers cluster includes people who 
arrived, visited a few pieces of content, and then left. Flowever, it also includes many learners who arrived 
late — in Week 1, in Week 2 or even later in the course, and who then did not sustain their engagement. 
In many cases, this late arrival may have indicated that they never intended to engage with the entire 
course. Flowever, it is likely that many found and joined the course after its start, or were unable to take 
part in its early days. This suggests that the time-bounded nature of the course, with a cohort working 
through the material together week by week, may have discouraged participation by late arrivals. 


The Strong Starters appear, in the FutureLearn datasets, to be distinct from the Samplers. On average, 
they completed the first week, they made one or two comments, and many of them looked beyond the 
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first week's materials. These learners did more than simply sample content, but were not able to sustain 
their engagement with the course. 

The clusters of Strong Starters, Returners, Mid-way Dropouts, and Nearly There are chiefly distinguished 
by the time period over which their members engaged with the course. Strong Starters engaged for a 
week. Returners for a couple of weeks. Mid-way Dropouts for about half the course, and Nearly There for 
all but the last week or two. Not surprisingly, an increase in time spent on the course was associated with 
a rise in the mean proportion of the course visited by group members and a rise in the mean number of 
comments contributed. 

The two clusters of Late Completers and Keen Completers can be considered as a pair, because they 
include the learners who have engaged with the majority of the material and all the assessments and who 
are therefore classified by FutureLearn as Fully Participating learners. The main characteristic that 
differentiates the two clusters is the number of comments posted. All the clusters considered previously 
in the analysis averaged about one post per person per week of engagement. On average. Samplers 
engaged for less than a week and posted less than one comment. Strong Starters participated for just over 
a week and posted just over one comment. Returners participated for a little over two weeks and posted 
just over two comments. Learners in the Nearly There cluster engaged for four to six weeks and posted 
around five comments. The Late Completers cluster also broadly followed this one-comment-per-week 
average. Flowever, the Keen Completers posted twice as much as those clusters, averaging well over two 
comments per week. 

It seems, then, that there were two dominant approaches to completing the course. The more popular 
approach (there were consistently more Keen Completers than Late Completers) was to engage fully. Keen 
Completers did almost everything, every week, and visited almost every step of the course. The Late 
Completers group did not engage with every aspect of the course: they commented less, and they were 
more frequently late in submitted assessments. So engaging actively with the comments is associated 
with a more extensive engagement with the course materials and assessment than is apparent in other 
clusters. 

4.1 Exploring the Clusters in More Detail 

A subsequent study, reported at EC-TEL 2015 (Ferguson et al., 2015), explored these clusters in more 
detail, examining learner engagement with MOOCs run by different universities on the FutureLearn 
platform. That study found that the seven main clusters identified in this paper were consistently found 
on seven- to eight-week MOOCs that support engagement with content, assessment, and discussion. 

These studies showed that the process of developing a "simple and scalable categorization to target 
interventions" (Kizilcec, Piech, & Schneider, 2013) is not as straightforward as it originally seemed. Studies 
of engagement suggest, perhaps unsurprisingly, that all MOOCs include learners who arrive, look around 
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a little, and then leave, as well as learners who arrive, engage with everything, and stay until the end. 
However, a large percentage of learners (37% in MOOCl, for example) do not fit these patterns. How 
these learners behave depends on the pedagogy of the course, and on how it is designed. A variation in 
course length, as with MOOC2, introduces new clusters, as does a variation in assessment patterns, as 
was the case with MOOC3. 

If engagement patterns shift with pedagogy and learning design, do they also shift as educators become 
more experienced and learners grow more familiar with learning in MOOCs? To answer this question, we 
returned to MOOCl and MOOC4 — the eight-week MOOCs with assessment points each week — and 
studied both the first and the second presentation of each of these courses. 

5 STUDY TWO: IDENTIFYING PATTERNS OF ENGAGEMENT ACROSS TIME 

5.1 Study Two: Dataset 

Table 4 (below) provides a brief overview of the dataset for Study 2, the two presentations of MOOCl and 
MOOC4. This overview supplements, and to some extent repeats, the information provided about these 
two MOOCs in Table 1 above. The figures for gender and for previous experience of online courses and 
MOOCs are taken from responses to the start-of-course survey for each presentation. The number of 
participants indicates the number of individuals — both educators and learners — who appeared in our 
dataset because they were active after the start date of the course. People who were active before the 
start date were removed from the dataset entirely, because the only people with access at that point 
were educators or FutureLearn staff. Fully Participating learners marked a majority of the course steps 
complete and completed all the assessments. 

Although the two presentations of each course were run less than a year apart, there is a change in 
participants' reported previous experience. In both cases, experience of courses run partly or solely online 
rises by two percentage points (n.s. for either). At the same time, previous experience of MOOCs rises by 
six to eight percentage points (p < 0.01 for MOOCl, p < 0.001 for MOOC2). This implies that the second 
presentation of MOOC4 had around 780 more participants with experience of MOOCs than it did on its 
first presentation, while the second presentation of MOOC4 had around 1,380 more participants with 
experience of MOOCs than it did on its first presentation. 

5.2 Study Two: Method 

For Study Two, we followed the method outlined in Section 3.2 (above). Individuals' engagement each 
week was scored using the values set out in Table 2 (above). Values assigned for different types of activity 
were totalled to assign each individual an engagement score ranging from 0 to 11 each week. The 
engagement scores for each of the eight weeks of the course were used to produce an engagement profile 
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containing eight values for each learner. The /f-means algorithm was used as before to extract seven 
clusters from these engagement profiles as an eight-dimensional vector for each learner. 


Table 4: Overview of dataset for Study Two 



MOOCl 

first 

MOOCl 

second 

MOOC4 

first 

MOOC4 

second 

Subject area 

Physical 

Sciences 

Physical 

Sciences 

Business 

Business 

Start month 

Mar 2014 

Feb 2015 

May 2014 

Mar 2015 

M 

51% 

50% 

35% 

36% 

F 

48% 

50% 

65% 

64% 

Taken online course before 

65% 

67% 

55% 

57% 

Taken MOOC before 

48% 

54% 

32% 

40% 

Participants 

5,069 

5,953 

9,778 

11,264 

Fully Participating 

1,548 

1,304 

1,416 

1,240 

Participation Rate 

31% 

30% 

14% 

11% 


5.3 Study Two: Analysis of Clusters 


The seven clusters for the second presentation of MOOCl and l\/IOOC4 were found on manual inspection 
to align very closely with those identified in Study One. Quantifying this similarity in the absence of an 
independent output variable is far from straightforward. As a simplistic but hopefully comprehensible 
approach, we calculated a series of descriptive statistics to facilitate comparison. 


Size: What percentage of learners was included in the cluster? Was any cluster consistently larger or 
smaller than the others? 


Steps: Each FutureLearn MOOC is divided into steps. A mean number of steps visited for each cluster was 
calculated by totalling all step visits and dividing by the number of learners in the cluster. This mean 
number was then compared with the total number of steps in the MOOC. 

Comments: What percentage of learners in the cluster added at least one comment to the MOOC 
discussion? 


Assessment: What percentage of learners in the cluster ever completed the first assessment? How many 
assignments were completed by at least 90% of the cluster? 

Considering these four areas enabled a detailed comparison of the attributes of each cluster. 

5.3.1 Study Two: Samplers 

Samplers (Table 5), the least engaged learners, make up the biggest cluster in each presentation of these 
two MOOCs. On average, they very consistently visited 4-5% of the steps. However, more than a fifth of 
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them visited only one step and left after their first visit. This cluster also contains a very high percentage 
of latecomers, with more than a third first visiting the course after its first week. Relatively few 
commented and few did the first — or any — assessment. 


Table 5: Study Two — Samplers 



MOOCl 1st 

MOOCl 2nd 

MOOC4 1st 

MOOC4 2nd 

Cluster 

characteristics 

Size of cluster 

39% 

’"47% 

56% 

56% 

Over a third 

Mean % steps visited 

5% 

5% 

5% 

*4% 

5%-10% 

% Visit only one step 

24% 

"*32% 

24% 

"’27% 

> 20% 

% Join after week 1 

27% 

"’34% 

39% 

’’’35% 

>25% 

% Leave a comment 

15% 

”’ll% 

13% 

12% 

< 20% 

% Do first assessment 

4% 

5% 

13% 

’"7% 

=<15% 

% Do any assessment 

6% 

6% 

15% 

"’io% 

=<15% 


* p < 0.05, ** p < 0.01, *** p < 0.001 between 1^* and 


5.3.2 Study Two: Strong Starters 

The Strong Starters (Table 6) are a smaller cluster than the Samplers (Table 5), but they are much more 
engaged. They visit more steps, a lot of them comment, and all of them complete the first assessment. 
However, few of them go on to do more assessments. 


Table 6: Study Two — Strong Starters 



MOOCl 1st 

MOOCl 2nd 

MOOC4 1st 

MOOC4 2nd 

Cluster 

characteristics 

Size of cluster 

11% 

12% 

10% 

"l4% 

8%-14% 

Mean % steps visited 

17% 

17% 

12% 

11% 

10%-20% 

% Visit only one step 

37% 

"’27% 

35% 

’"28% 

20%-40% 

% Do first assessment 

100% 

100% 

100% 

100% 

100% 

% Do >1 assessment 

5% 

6% 

7% 

*5% 

<10% 


* p< 0.05, ** p< 0.01, *** p < 0.001 between l'‘ and 2"“ 


5.3.3 Study Two: Returners 

The Returners (Table 7) make up a small, but fairly active cluster. They engage with a lot of the content, 
although none of them looks at it all. Most of them do the first assessment and go on to do another 
assessment, and a lot of them engage by commenting. 
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Table 7: Study Two — Returners 



MOOCl 

1st 

MOOCl 

2nd 

MOOC4 

1st 

MOOC4 

2nd 

Cluster 

characteristics 

Size of cluster 

6% 

8% 

7% 

8% 

6%-8% 

Mean % steps visited 

33% 

36% 

24% 

26% 

20%-50% 

% Visit all steps 

0% 

0% 

0% 

0% 

0% 

% Leave a comment 

39% 

35% 

23% 

*"32% 

>20%-60% 

% Do first assessment 

95% 

96% 

93% 

94% 

>90% 

% Do > 1 assessment 

95% 

96% 

91% 

91% 

>90% 


*** p < 0.001 between 1^' and 2"^ 


5.3.4 Study Two: Midway Dropouts 

The Midway Dropouts (Table 8) make up another small cluster. Almost all of them do the first assessment, 
and most do three or more assessments. About half of them comment at least once, and they view about 
half of the course content. 


Table 8: Study Two — Midway Dropouts 



MOOCl 1st 

MOOCl 2nd 

MOOC4 1st 

MOOC4 2nd 

Cluster 

Characteristics 

Size of cluster 

6% 

7% 

7% 

6% 

6%-7% 

Mean % steps visited 

59% 

65% 

49% 

48% 

45%-65% 

% Leave a comment 

49% 

43% 

38% 

**46% 

35%-50% 

% Do first assessment 

98% 

97% 

96% 

***99% 

>95% 

% Do > 2 assessments 

97% 

99% 

96% 

*98% 

>90% 


* p< 0.05, ** p < 0.01, *** p < 0.001 between 1 * and 2"'* 


5.3.5 Study Two: Nearly There 

Nearly There (Table 9) is consistently the smallest cluster, made up of people who view most of the course 
content and do most of the assessment. They are also more likely than not to post a comment. 


Table 9: Study Two — Nearly there 



MOOCl 1st 

MOOCl 2nd 

MOOC4 1st 

MOOC4 2nd 

Cluster 

Characteristics 

Size of cluster 

6% 

5% 

5% 

4% 

4%-6% 

Mean % Steps visited 

76% 

***89% 

76% 

74% 

70%-90% 

% Leave a comment 

65% 

**53% 

48% 

*55% 

45%-65% 

% Do first assessment 

98% 

98% 

99% 

99% 

>95% 

% Do > 4 assessments 

99% 

***100% 

97% 

97% 

>90% 


* p< 0.05, ** p< 0.01, *** p < 0.001 between 1 ‘ and 2"“ 


5.3.6 Study Two: Late Completers 

In most cases, the engagement in clusters rises consistently. Each cluster described above has viewed 
more content than the last, included more people who comment than the last, and completed more 
assessments than the last. The Late Completers (Table 10) are an exception. They visit almost all the steps, 
they do almost all of the assessment, but the percentage that comment is relatively low. 
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Table 10: Study Two — Late completers 



MOOCl 1st 

MOOCl 2nd 

MOOC4 1st 

MOOC4 2nd 

Cluster 

Characteristics 

Size of cluster 

8% 

6% 

6% 

4% 

6%-8% 

Mean % steps visited 

92% 

95% 

95% 

92% 

90%-97% 

% Leave a comment 

44% 

38% 

42% 

*36% 

35%-45% 

% Do first assessment 

96% 

97% 

98% 

97% 

>95% 

% Do >6 assessments 

93% 

93% 

95% 

*'*89% 

>90% 


* p < 0.05, ** p < 0.01, *** p < 0.001 between 1^* and 2"'' 


5.3.7 Study Two: Keen Completers 

The Keen Completers (Table 11) can be considered ideal MOOC students from an educator's perspective. 
They visit all, or almost all, of the content. They do all, or almost all of the assessment. Two-thirds of them 
add to the course by commenting. They represent most, but not all, of the learners classified as "Fully 
Participating" and therefore eligible for a FutureLearn Statement of Participation. 


Table 11: Study Two — Keen completers 



MOOCl 1st 

MOOCl 2nd 

MOOC4 1st 

MOOC4 2nd 

Cluster 

characteristics 

Size of cluster 

23% 

”*15% 

9% 

8% 

8%-23% 

Mean % steps visited 

95% 

’**100% 

98% 

99% 

95%-100% 

% Leave a comment 

71% 

*66% 

68% 

67% 

65%-75% 

% Do first assessment 

100% 

100% 

100% 

100% 

>98% 

% Do >7 assessments 

99% 

99% 

100% 

100% 

>90% 


* p < 0.05, ** p < 0.01, *** p < 0.001 between 1^* and 2"'' 


5.4 Study Two — Discussion 

Are the clusters effectively the same? This is difficult to quantify. The clusters are clearly not identical: 
some of the descriptive statistics in Tables 5-11 are statistically significantly different from one 
presentation to the next at a high level of significance. But are those differences meaningful? As an 
extreme, 99% of learners in the Nearly There cluster first presentation of MOOCl completed more than 
four assessments, compared to 100% in the second (see bottom Table 9), and this difference is significant 
at greater than the 0.001 level. It seems hard to argue that this is a meaningful difference between clusters 
of learner behaviours on subsequent presentations of the same MOOC. 

Standard hypothesis testing, as we have carried out here, seeks to reject the null hypothesis of no 
difference between the two groups. This is not the same as testing whether there is no difference: that is 
the domain of testing for equivalence. To test for equivalence, one has to first define the range of results 
that would be considered effectively the same, and then test whether the results fall inside some 
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confidence interval based on that range. The "cluster characteristics" column is our suggestion for a range 
that is effectively the same — and all the data fall within this range. 

For instance, the precise percentage of learners in the Samplers cluster (Table 5) who leave a comment is 
different in a statistically significantly way from one presentation of a MOOC to a subsequent one (15% 
on MOOCl's first presentation, 11% on the second). However, it is fairly low in all cases (<20%), and this 
(and other characteristics) distinguishes this cluster from the other clusters. The clusters are practically 
similar between MOOCs and between one presentation of a MOOC and the subsequent one. 

Unfortunately, as already discussed, the dataset available for these studies does not contain any outcome 
measures (such as assessment results or survey responses) that would provide an independent 
comparison. 

The similarity in the descriptive statistics from the more than 30,000 learners who engaged with the two 
presentations of MOOCl and MOOC4 indicates that these seven clusters are stable across presentations. 
It is not possible to test this directly, as the assessment and survey data cannot be aligned with activity 
data. However, there is no evidence that these clusters change meaningfully as learners gain more 
experience in online learning. 

6 CONCLUSION 

6.1 Limitations of this Approach to Ciuster Anaiysis 

This work has followed and extended the original methodology of Kizilcec et al. It has yielded some 
potentially useful insights for practice and for future research (see subsequent sections). However, there 
are significant limitations to the approaches taken by both this work and the work it builds on. 

Both approaches entailed significant processing and data reduction and choices around the cluster 
analysis. One significant practical advantage of coding weekly behaviour in to a single measure is that it 
enables analysts to inspect individual patterns at a glance in a way impossible if the direct logfile data is 
used. However, this makes data unavailable for clustering: for instance, it would be impossible to 
distinguish learners who tended to work in one burst a week from learners who studied in a more spread- 
out pattern. The approach taken to the cluster analysis in both approaches is rather simplistic. It is not 
obvious that k-means is the best clustering algorithm in this context — not least because it requires what 
may be an arbitrary decision to be made by the analyst about how many clusters to extract. 

It seems highly likely that the patterns that emerged in the final clusters reflect, at least to some degree, 
the choices made in this process. Kizilcec et al. coded weekly behaviour based on whether or not learners 
engaged with content or did the assessment, and compressed this in to a single dimension before 
clustering: they found four clusters that reflected the four logical possibilities (Sampling — low 
engagement with content; Auditing — high engagement with content; Disengaging — low engagement 
with assessments; Completing — high engagement with assessment). The adapted method in this paper 
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added data about commenting, and found clusters where commenting behaviour was a distinguishing 
feature. Rather than compressing the weekly data in to a single dimension, this paper used a dimension 
for each week, and duly found clusters where behaviour changed over the weeks. 

That is not to say that these patterns do not exist in the data: they almost certainly do. However, they do 
not emerge from the data without influence from the choices made in the analysis. Accordingly, we wish 
to be very modest about the clusters from our own analysis. They are far from a correct, complete, or 
robust analysis of the participation patterns of learners in a MOOC. We claim only that they provide an 
informative view of learner behaviour, which has some potentially valuable implications for practice. 
However, two patterns of behaviour do seem to emerge robustly from cluster analysis of learner 
behaviour on MOOCs: samplers, who visit only briefly; and completers, who fully engage with the course. 

6.2 Implications for Practice 

The key characteristic of learning analytics, as distinct from the general category of quantitative 
educational research, is a direct concern with improving learning and learning environments. "The key 
step is "closing the loop" by feeding back [...] to learners through one or more interventions" (Clow, 2012). 
How can this analysis achieve that? 

The clusters identified here can help inform a range of strategies for intervention and improvement. As 
these clusters have now been found on a range of presentations of MOOCs from different universities and 
covering different subjects, educators can be confident that they are stable as long as the learning design 
(eight-week courses with weekly assessment) and pedagogy (conversational learning) remain constant. 

Providing previews of course material would allow Samplers to make a more informed decision about 
whether to register in the first place. 

The four clusters in which learners engage solidly but fail to complete — the Strong Starters, Returners, 
Mid-way Dropouts, and particularly the Nearly There cluster — are precisely those learners that educators 
would want to focus on to improve completion. This data analysis suggests that there are several points 
at which learners may leave a course, and these may be dealt with in different ways. 

Discussion steps set up for latecomers could be used to support those who fall behind at the start, while 
sign-up pages could draw attention to the difficulties encountered by learners who move through the 
course out of step with the cohort. These pages might suggest that potential learners return later and 
register for a subsequent presentation. Free-text survey comments on FutureLearn MOOCs that have run 
more than once show that some learners return to subsequent presentations in order to finish working 
through the material with a group — even though they already have access to the same content because 
they registered on the original presentation. 
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The course week is an artificial distinction that imposes a weekly format on the learning design of the 
course. In doing so, it makes the decision to move on to the next week of the course appear more 
significant than the decision to move on to the next step. Changes to learning design could provide bridges 
between course weeks, stressing links between these weeks, and pointing learners forward. This might 
reduce the drop-off experienced by these courses, particularly at the ends of Week 1 and Week 2. 

The two most clearly successful clusters — the Late Completers and the Keen Completers — also afford 
actionable insights. In the context of FutureLearn, at least, they appear to provide some small degree of 
vindication of the extensive and structured use of discussion comments. 

6.3 Implications for Future Research 

These clusters could be used to support qualitative research into learner engagement. In particular, it 
would be useful for practice to identify what prompts Keen Completers to engage to such an extent, and 
what factors influence drop out by the Nearly There cluster. 

However, these results suggest that this approach to clustering data from learners on MOOCs has been 
pursued as far as is defensible. Neither the Kizilcec et al. approach nor the adaptation of it in this paper 
appears to be particularly robust. Future work should explore more rigorous and sophisticated 
approaches to analyzing learner behaviour, including less reductive data preparation and coding, other 
clustering methods (e.g., hierarchical clustering approaches), similarity measures other than Euclidean 
distance (e.g., Levenshtein distance on textual profiles), and so on. Other data may well prove useful. For 
instance, it is well known that simply counting forum contributions can be misleading if one is interested 
in the contribution to learning. A more robust set of patterns would be potentially very useful for 
practitioners to target on-course support. 

This work has explored patterns of individual learner behaviour within a MOOC, and on subsequent 
presentations of the same MOOC. With a more robust classification system for learner behaviour, it would 
be interesting to explore learner behaviour in different MOOCs, which might shed light on the stability or 
otherwise of such behaviours and on what associations there may be between MOOC characteristics and 
learner behaviours. 

6.4 Final Remarks 

As open offerings, for some values of "open," MOOCs afford a wide range of engagement patterns. Two 
have emerged consistently from this work and previous research: the sampling behaviour employed by 
people who visit a course briefly, and the fully engaged behaviour employed by learners who engage with 
all aspects of a course. The studies reported here suggest that both these and other patterns of 
engagement remain stable across presentations of a course, when key elements of pedagogy and learning 
design remain constant. 
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